A Hybrid End-to-End Spatio-Temporal Attention Neural Network with Graph-Smooth Signals for EEG Emotion Recognition

Recently, physiological data such as electroencephalography (EEG) signals have attracted significant attention in affective computing. In this context, the main goal is to design an automated model that can assess emotional states. Lately, deep neural networks have shown promising performance in emotion recognition tasks. However, designing a deep architecture that can extract practical information from raw data is still a challenge. Here, we introduce a deep neural network that acquires interpretable physiological representations by a hybrid structure of spatio-temporal encoding and recurrent attention network blocks. Furthermore, a preprocessing step is applied to the raw data using graph signal processing tools to perform graph smoothing in the spatial domain. We demonstrate that our proposed architecture exceeds state-of-the-art results for emotion classification on the publicly available DEAP dataset. To explore the generality of the learned model, we also evaluate the performance of our architecture towards transfer learning (TL) by transferring the model parameters from a specific source to other target domains. Using DEAP as the source dataset, we demonstrate the effectiveness of our model in performing cross-modality TL and improving emotion classification accuracy on DREAMER and the Emotional English Word (EEWD) datasets, which involve EEG-based emotion classification tasks with different stimuli

Towards adaptive brain-computer interfaces: statistical inference for mental state recognition

Brain-computer interface (BCI) systems aim to establish direct communication channels between the brain and external devices. The primary motivation is to enable patients with limited or no muscular control, including amyotrophic lateral sclerosis (ALS) and stroke patients, to use computers or other devices by automatically interpreting their intent based on the measured brain electrical activity. Furthermore, enabling healthy individuals to use BCI systems as an additional communication channel in certain human computer interaction systems is also a current topic of interest. Current experimental BCI systems are trained in a supervised fashion and then evaluated during test sessions. With increasing demands for daily and long-term use of BCIs in real-life applications such as in semi-autonomous cars, BCIs have been tested on longer sessions in which researchers have observed considerably lower performance of trained systems. This is believed to be caused by the nonstationary nature of the electroencephalographic (EEG) signals. As a result, semi-supervised adaptation of BCI systems based on test data has emerged as a new research domain. One of the main reasons underlying the nonstationarity of signals involves changes in the users’ cognitive states such as the cognitive load, alertness, attention, fatigue, boredom, and motivation. However, dynamically extracting information about such cognitive states from EEG signals and using that to improve the performance of BCI systems is currently an open research problem. In this thesis, we tackle the highly complex problem of estimating the level of alertness and vigilance of users during execution of cognitive tasks. To identify the neural, EEGbased correlates of long-term task and response time consistency, we devise a series of experiments running the sustained attention to response task (SART). After proposing a novel adaptive scoring scheme for vigilance, we provide new evidence on the close relationship between intrinsic resting and task-related brain networks and develop models to predict consistency in tonic performance and response time using neural networks and feature relevance analysis from spatio-spectral features of resting-state EEG signals. Next, focusing on the imminent goal of predicting low and high vigilance intervals, we propose fully automated systems based on convolutional neural networks (CNNs) using phase locking value features as successful pre-trial predictors of phasic vigilance and performance consistency. In all of these contributions, we consider the personal vigilance traits and individual psychophysiological differences for modeling and detecting the extremely alert and drowsy trials in long and monotonous experiments, and enrich the literature with the evidence on spatio-spectro-temporal correlates of vigilant and consistent behavior. We then utilize Bayesian changepoint models for sequential inference and detection of instants at which continuous vigilance levels of users enter a new phase. We demonstrate the success of our online and offline vigilance models in detecting changepoints from both the SART datasets collected in our lab and driving datasets that contain vigilance labels. Finally and as the highlight of this thesis, we hypothesize that the underlying vigilance levels affect users’ reaction time and thus the ability to focus and engage in motor imagery BCI paradigms. We then introduce an adaptive alertness-aware MI classification system for motor imagery BCI that uses a series of novel unsupervised learning schemes for labeling trial vigilance levels during training and test sessions, and leads to a method with full adaptation in both feature extraction and training of its classifier parameters. Three different versions of this adaptive classification approach are introduced that are trained differently on trials labeled with low vigilance levels by our various vigilance clustering schemes. We report improvements in the overall test accuracy of adaptive versions with respect to the original, non-adaptive baseline for our own SPIS MI-BCI dataset and the BCI Competition IV Dataset 2a. A number of datasets collected in our BCI laboratory are uploaded to a public repository at https://github.com/mastaneht

Prediction of motor imagery performance based on pre-trial spatio-spectral alertness features

Electroencephalogram (EEG) based brain-computer interfaces (BCIs) enable communication by interpreting the user intent based on measured brain electrical activity. Such interpretation is usually performed by supervised classifiers constructed in training sessions. However, changes in cognitive states of the user, such as alertness and vigilance, during test sessions lead to variations in EEG patterns, causing classification performance decline in BCI systems. This research focuses on effects of alertness on the performance of motor imagery (MI) BCI as a common mental control paradigm. It proposes a new protocol to predict MI performance decline by alertness-related pre-trial spatio-spectral EEG features. The proposed protocol can be used for adapting the classifier or restoring alertness based on the cognitive state of the user during BCI applications

Prediction of reaction time and vigilance variability from spatio-spectral features of resting-state EEG in a long sustained attention task

Resting-state brain networks represent the intrinsic state of the brain during the majority of cognitive and sensorimotor tasks. However, no study has yet presented concise predictors of task-induced vigilance variability from spectro-spatial features of the resting-state electroencephalograms (EEG). In this study, ten healthy volunteers have participated in fixed-sequence, varying-duration sessions of sustained attention to response task (SART) for over 100 minutes. A novel and adaptive cumulative vigilance scoring (CVS) scheme is proposed based on tonic performance and response time. Multiple linear regression (MLR) using feature relevance analysis has shown that average CVS, average response time, and variabilities of these scores can be predicted (p < 0.05) from the resting-state band-power ratios of EEG signals. Cross-validated neural networks also captured different associations for narrow-band beta and wide-band gamma and differences between the high- and low-attention networks in temporal regions. The proposed framework and these first findings on stable and significant attention predictors from the power ratios of resting-state EEG can be useful in brain-computer interfacing and vigilance monitoring applications

Emotionality of Turkish language and primary adaptation of affective English norms for Turkish

Emotional load assessment of the written words has gained considerable interest in psycholinguistics, semantics, and analysis of psychophysiological and electrophysiological correlates of emotional processing. Considering the lack of a publicly available database with affective ratings of contemporary verbal stimuli obtained from native Turkish speakers, we present the affective norms for two datasets of Turkish words carefully adapted from the Affective Norms for English Words (ANEW) database. The valence and arousal ratings are obtained from 61 college-aged participants for 127 highly arousing, emotionally-loaded words in the Adapted Turkish Affective List (ATAL). The ATAL ratings show a tendency of classifying fewer words as positive compared to the original list of stimuli, significantly higher arousal levels for positively rated Turkish stimuli compared to the negative and neutral words, and more congruence in arousal levels of positively exciting words. For the medium to high arousing 508 words in the Expanded Turkish Affective List (ETAL) that cover the whole 9-point spectrum of the valence dimension, 136 Turkish respondents from a wider age, education, and occupation background show higher excitability towards highly unpleasant words. Strong cross-linguistic correlations of +0.968 between the valence ratings of ANEW and ATAL and +0.878 for ANEW and ETAL demonstrate the ease of transferring and perceiving the valence levels across English and Turkish. The medium correlation of roughly +0.450 between the English and Turkish arousal ratings account for lower excitation levels perceived by the native Turkish speakers and indicate the arousal dimension is similar to familiarity and originality in exhibiting more variations between different cultures. These findings demonstrate that this expanded database of partial affective normative ratings can be used as the ground truth for emotional and neurocognitive assessments, and that the presented methodology can be utilized for developing a comprehensive Turkish affective lexicon. The utilized word selection criteria also enable a cross-cultural analysis of adapted words in Turkish and other languages. Detailed normative ratings of this Turkish adaptation are included in the supplementary materials